Add user-facing message field to violation output by oshorefueled · Pull Request #71 · TRocket-Labs/vectorlint

oshorefueled · 2026-03-04T15:00:03Z

Why

Terminal output showed analysis — the model's internal reasoning — as the user-facing violation message. This made findings feel verbose and opaque, as if written for an internal reviewer rather than the user.

What

Add a message field to the LLM violation schema: required, under 15 words, written directly to the user with no rule references
Display message instead of analysis in the terminal
Expose analysis in native JSON output (--output=json) so both fields are available to consumers
Add directive guidance telling the model that message is shown to the user and analysis is not

Scope

In scope

message field added to both buildCheckLLMSchema() and buildJudgeLLMSchema() violation objects
message added to all corresponding TypeScript violation types (JudgeLLMResult, CheckLLMResult, JudgeResult, CheckItem, CheckResult)
Terminal display updated to use v.message
analysis field added to the Issue interface and included in --output=json violation objects
Directive updated to distinguish the two fields

Out of scope

Changes to analysis field content or behavior
Scoring, confidence, or PAT gate logic
Vale JSON and RdJSON output formats (unaffected — they map to fixed external schemas)

Behavior impact

Terminal output: violation messages are now sourced from message (short, user-facing) instead of analysis (internal reasoning). Users will see shorter, cleaner findings.
--output=json: message is a new required field and analysis is now included on every violation object. Both fields are additive — existing consumers will not break.
--output=vale-json / --output=rdjson: no change.
Token usage: the LLM now generates one additional required string field per violation.

Risk

Low. The terminal display change is the only behavioral difference for end users.
JSON consumers that enumerate violation fields will see two new fields (message, analysis); both are additive.
No scoring, config, or provider changes.

How to test / verify

Checks run

npm run test:run — 228 tests, 32 files, all passed
npm run build — clean build, no TypeScript errors

Manual verification

Run vectorlint on a file with known violations: npm run dev -- <file>
Confirm terminal output shows short findings (≤15 words, no rule references)
Run with --output=json and confirm each violation object includes both message and analysis
Run with --output=vale-json and confirm output is unchanged

Rollback

Revert to main. No migrations, config changes, or external dependencies introduced.

Summary by CodeRabbit

New Features
- Issues can include optional analysis metadata in JSON output for richer reporting.
- Violation reports now carry a concise, user-facing message summary for clearer terminal/API output.
Tests
- Added tests to validate the presence and formatting (<=15 words) of the message field in violation reporting schemas.

- Add `message: string` to JudgeLLMResult violation array - Add `message: string` to CheckLLMResult violation array - Add `message: string` to JudgeResult criteria violations array - Add `message?: string` (optional) to CheckResult violations array - Add `message?: string` (optional) to CheckItem type - Add required-array assertions for message in prompt-schema tests

coderabbitai · 2026-03-04T15:00:26Z

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4379f097-9bbf-4551-bac5-02d7571df2bc

📥 Commits

Reviewing files that changed from the base of the PR and between f7b4599 and 18e923e.

📒 Files selected for processing (1)

src/cli/types.ts

📝 Walkthrough

Walkthrough

Adds an optional analysis field and a required user-facing message field to the violation/reporting pipeline: type definitions, LLM schemas, orchestrator reporting, and JSON output are updated so per-violation message is used for UI text and analysis is propagated as internal metadata.

Changes

Cohort / File(s)	Summary
Type definitions `src/cli/types.ts`, `src/output/json-formatter.ts`	Added `analysis?: string` to `ReportIssueParams` and `Issue`/`JsonIssue` shapes; added `message?: string` to violations in process params to reflect LLM payload shape.
Orchestrator / CLI `src/cli/orchestrator.ts`	Added `reportIssue(params: ReportIssueParams)` and threaded `analysis` through issue reporting; switched per-row text source from `v.analysis` → `v.message` when building violation output.
LLM prompts & schemas `src/prompts/schema.ts`, `src/prompts/directive-loader.ts`	Extended Check and Judge schemas so violations include a required `message: string` (user-facing, <=15 words) and preserved `analysis` as internal metadata; directive loader documents `message` as UI-facing.
Output formatting `src/output/json-formatter.ts`	JSON issue shape now optionally includes `analysis` when provided by the orchestrator/violations.
Tests `tests/prompt-schema.test.ts`	Added tests asserting `message` exists on check/judge violation schema items, has type string and the specified description, and that `message` is required.

Sequence Diagram(s)

sequenceDiagram
  participant LLM as LLM (Check/Judge)
  participant Orch as Orchestrator
  participant Formatter as JSONFormatter
  participant Out as Output (file/stdout)

  LLM-->>Orch: return violations[] { rule, match, message, analysis? }
  Orch-->>Orch: process violations (use v.message for UI row)
  Orch-->>Formatter: build JsonIssue(s) including analysis? and message
  Formatter-->>Out: emit JSON array of issues (issue.message, issue.analysis?)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Remove zero-violation JSON noise and restore concise LLM guidance #70: Modifies linter JSON/reporting paths and schema for analysis/message fields—strongly related to the same reporting/schema changes here.
feat: add drop-in replacement fix field to lint output #58: Threads an optional per-violation field through reportIssue, types, JSON formatter, and schemas—matches the propagation pattern used in this change.

Poem

🐰 A hop, a nudge, a careful clue,

"Message" speaks loud, "analysis" stays true.
I nibble schema, stitch it neat,
Two fields now dance where issues meet.
Hooray — the rabbit guards the sheet! 🌿

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately and concisely summarizes the main change: adding a user-facing message field to violation output, which is the core objective across schema updates, display changes, and JSON output modifications.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings (stacked PR)
📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch enh/user-message

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

oshorefueled added 7 commits March 4, 2026 15:33

feat: add message field to check and judge LLM schemas

6b88da8

feat: display message field instead of analysis in terminal output

ea04e7b

docs: add directive guidance distinguishing message from analysis

410e52b

fix: remove unnecessary analysis fallback from rowSummary

d3556ba

fix: use 'user' instead of 'document author' in message field guidance

23379b8

feat: include analysis field in native JSON output

f7b4599

fix: add message to ProcessViolationsParams violations type

18e923e

oshorefueled merged commit 27097c3 into main Mar 4, 2026
2 checks passed

oshorefueled deleted the enh/user-message branch March 4, 2026 15:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add user-facing message field to violation output#71

Add user-facing message field to violation output#71
oshorefueled merged 8 commits intomainfrom
enh/user-message

oshorefueled commented Mar 4, 2026 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Mar 4, 2026 •

edited

Loading

Review failed

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Poem

❌ Failed checks (1 warning)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

oshorefueled commented Mar 4, 2026 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Scope

Behavior impact

Risk

How to test / verify

Rollback

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related PRs

Poem

❌ Failed checks (1 warning)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

oshorefueled commented Mar 4, 2026 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Mar 4, 2026 •

edited

Loading